Feature: support rectilinear chunk grid extension #3534
Conversation
github-actions
bot
added
the
needs release notes
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3534 +/- ##
==========================================
+ Coverage 61.86% 62.18% +0.32%
==========================================
Files 85 85
Lines 10111 10604 +493
==========================================
+ Hits 6255 6594 +339
- Misses 3856 4010 +154
🚀 New features to boost your workflow:
|
…ature/rectilinear-chunk-grid
github-actions
bot
removed
the
needs release notes
| when you need to align chunks with existing data partitions. | ||
|
|
There was a problem hiding this comment.
| when you need to align chunks with existing data partitions. | |
| when you need to align chunks with existing data partitions. | |
| The specification for this chunking scheme can be found [here](https://github.com/zarr-developers/zarr-extensions/tree/main/chunk-grids/rectilinear/). |
This link doesn't resolve yet but it will when the spec is merged.
| With variable chunking, the standard `.chunks` property is not available since chunks | ||
| have different sizes. Instead, access chunk information through the chunk grid: |
There was a problem hiding this comment.
I think it would be better if .chunks just had a different type (tuple of tuples of ints)
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class RectilinearChunkGrid(ChunkGrid): |
There was a problem hiding this comment.
thoughts on just calling this class Rectilinear, and renaming the RegularChunkGrid to Regular? We could keep around a RegularChunkGrid class for compatibility. But I feel like people know these are chunk grids when they import them
| ) | ||
|
|
||
| @cached_property | ||
| def _cumulative_sizes(self) -> tuple[tuple[int, ...], ...]: |
src/zarr/testing/strategies.py
Outdated
| chunk_shapes_rle = [ | ||
| [[c, r] for c, r in zip(draw(dim_chunks), draw(repeats), strict=True)] | ||
| for _ in range(ndim) | ||
| ] | ||
| return RectilinearChunkGrid(chunk_shapes=chunk_shapes_rle) |
| @given(data=st.data()) | ||
| async def test_basic_indexing(data: st.DataObject) -> None: | ||
| zarray = data.draw(simple_arrays()) | ||
| @given(data=st.data(), zarray=st.one_of([simple_arrays(), complex_chunked_arrays()])) |
There was a problem hiding this comment.
Because the search space for the standard arrays strategy is so large, i made a different one complex_chunked_arrays that purely checks different chunk grids
with simple_arrays() we are only spending 10% of our time trying RectilinearChunkGrid so using this approach. We should boost number of examples too.
| 2. **Not compatible with sharding**: You cannot use variable chunking together with | ||
| the sharding feature. Arrays must use either variable chunking or sharding, but not both. |
There was a problem hiding this comment.
I hope this is a temporary limitation! There's a natural extension of rectilinear chunk grids to rectilinear shard grids.
There was a problem hiding this comment.
4 participants
(Marking this as a draft for now)
closes: #1595
replaces: #1483
xref: zarr-developers/zarr-extensions#25
TODO:
docs/user-guide/*.mdchanges/